Improving the dataloader tests. #6187

drivanov · 2023-08-22T00:42:12Z

Description

Elimination or suppression of the following warnings that appear in tests implemented in
dataloading/test_dataloader.py:

tests/python/pytorch/dataloading/test_dataloader.py: 93 warnings
  /usr/local/lib/python3.10/dist-packages/dgl/dataloading/dataloader.py:1149: DGLWarning: Dataloader CPU affinity opt is not enabled, consider switching it on (see enable_cpu_affinity() or CPU best practices for DGL [https://docs.dgl.ai/tutorials/cpu/cpu_best_practises.html])

These warnings are eliminated when the dataloader is created for num_workers > 0 or suppressed for num_workers = 0 .

Checklist

Please feel free to remove inapplicable items for your PR.

I've leverage the tools to beautify the python and c++ code.
The PR is complete and small, read the Google eng practice (CL equals to PR) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
All changes have test coverage
Code is well-documented
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

dgl-bot · 2023-08-22T00:42:42Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-22T00:42:54Z

Commit ID: 7777165

Build ID: 1

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-08-22T03:16:17Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-22T03:16:29Z

Commit ID: aa5159593e175236ad9b099433822e4b56278449

Build ID: 2

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

Rhett-Ying · 2023-08-22T03:51:05Z

@dgl-bot

dgl-bot · 2023-08-22T04:35:59Z

Commit ID: 1f23894128065a22d41250213ca5bf89cec8ccb7

Build ID: 3

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

dgl-bot · 2023-08-22T05:18:22Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

Rhett-Ying · 2023-08-22T05:18:27Z

Commit ID: 1f23894128065a22d41250213ca5bf89cec8ccb7

Build ID: 3

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

@drivanov pls address it.

dgl-bot · 2023-08-22T05:18:34Z

Commit ID: 9fcdb6e

Build ID: 4

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

…o test_dataloader

dgl-bot · 2023-08-22T16:29:10Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-22T16:29:23Z

Commit ID: 69a03c2

Build ID: 5

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-08-23T00:55:01Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-23T00:55:13Z

Commit ID: 4fc11c6

Build ID: 6

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-08-23T01:16:30Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-23T01:16:43Z

Commit ID: 30643c0

Build ID: 7

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

Rhett-Ying · 2023-08-23T01:17:48Z

@dgl-bot

dgl-bot · 2023-08-23T01:51:26Z

Commit ID: 30643c0

Build ID: 8

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

dgl-bot · 2023-08-24T21:08:11Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-24T21:08:24Z

Commit ID: 5b07edd

Build ID: 9

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

Rhett-Ying · 2023-08-25T00:35:37Z

@dgl-bot

dgl-bot · 2023-08-25T01:22:19Z

Commit ID: 5b07edd

Build ID: 10

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

dgl-bot · 2023-08-29T16:46:30Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-29T16:46:42Z

Commit ID: 5d3dba8

Build ID: 11

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

frozenbugs · 2023-08-30T05:00:48Z

@dgl-bot

dgl-bot · 2023-08-30T05:45:10Z

Commit ID: 1703a67f1536a6945e42242f61661a629b35a911

Build ID: 12

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

drivanov · 2023-08-30T17:29:11Z

Unfortunately, I can't reproduce these issues in the latest version of the NVidia container. Looks like torch\utils\data\dataloader.py has been updated and we are still using the old version. Let's wait until we start using the new version of this file. I changed the status of the current PR to "Draft".

dgl-bot · 2023-08-30T17:35:58Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-08-30T17:36:10Z

Commit ID: 7cd232b

Build ID: 13

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

…o test_dataloader

dgl-bot · 2023-09-05T18:29:50Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-09-05T18:30:03Z

Commit ID: 4520b1c023b804f57db15c64bfacef9567d14efe

Build ID: 14

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-09-05T18:30:54Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-09-05T18:31:06Z

Commit ID: 5b48c35

Build ID: 15

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

drivanov · 2023-09-05T18:37:48Z

It looks like the real reason for the failed tests that I couldn't reproduce in my Linux environment was some kind of multithreading bug that shows up only on Windows. I did make some changes and we will see if them will fix that problem.

frozenbugs · 2023-09-06T01:17:49Z

@dgl-bot

dgl-bot · 2023-09-06T01:55:17Z

Commit ID: 5b48c35

Build ID: 16

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

dgl-bot · 2023-09-06T16:45:20Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-09-06T16:45:32Z

Commit ID: 23f86c28d9f02dad313125b7f3a8e889cb4d6f90

Build ID: 17

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

Improving the dataloader tests.

7777165

Fixing lint problems

8217be1

Merge branch 'master' into test_dataloader

9fcdb6e

drivanov added 2 commits August 22, 2023 09:20

Adding comment

3f4a1c4

Merge branch 'test_dataloader' of https://github.com/drivanov/dgl int…

69a03c2

…o test_dataloader

Fixing test_edge_dataloader_excludes running on gpu

4fc11c6

Fixing lint problem.

30643c0

Merge branch 'master' into test_dataloader

5b07edd

Merge branch 'master' into test_dataloader

5d3dba8

frozenbugs self-requested a review August 30, 2023 05:00

drivanov marked this pull request as draft August 30, 2023 17:28

Merge branch 'master' into test_dataloader

7cd232b

drivanov added 2 commits September 5, 2023 11:18

Fixing a problem with the context that appeared in Windows.

3a51d83

Merge branch 'test_dataloader' of https://github.com/drivanov/dgl int…

1d0f76c

…o test_dataloader

drivanov marked this pull request as ready for review September 5, 2023 18:30

Merge branch 'master' into test_dataloader

5b48c35

drivanov marked this pull request as draft September 6, 2023 16:44

drivanov mentioned this pull request Feb 20, 2024

Improvement to the test_spot_target_excludes test. #6652

Draft

3 tasks

Improving the dataloader tests. #6187

Are you sure you want to change the base?

Improving the dataloader tests. #6187

Uh oh!

Conversation

drivanov commented Aug 22, 2023

Description

Checklist

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

Rhett-Ying commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

Rhett-Ying commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 22, 2023

Uh oh!

dgl-bot commented Aug 23, 2023

Uh oh!

dgl-bot commented Aug 23, 2023

Uh oh!

dgl-bot commented Aug 23, 2023

Uh oh!

dgl-bot commented Aug 23, 2023

Uh oh!

Rhett-Ying commented Aug 23, 2023

Uh oh!

dgl-bot commented Aug 23, 2023

Uh oh!

dgl-bot commented Aug 24, 2023

Uh oh!

dgl-bot commented Aug 24, 2023

Uh oh!

Rhett-Ying commented Aug 25, 2023

Uh oh!

dgl-bot commented Aug 25, 2023

Uh oh!

dgl-bot commented Aug 29, 2023

Uh oh!

dgl-bot commented Aug 29, 2023

Uh oh!

frozenbugs commented Aug 30, 2023

Uh oh!

dgl-bot commented Aug 30, 2023

Uh oh!

drivanov commented Aug 30, 2023

Uh oh!

dgl-bot commented Aug 30, 2023

Uh oh!

dgl-bot commented Aug 30, 2023

Uh oh!

dgl-bot commented Sep 5, 2023

Uh oh!

dgl-bot commented Sep 5, 2023

Uh oh!

dgl-bot commented Sep 5, 2023

Uh oh!

dgl-bot commented Sep 5, 2023

Uh oh!

drivanov commented Sep 5, 2023

Uh oh!

frozenbugs commented Sep 6, 2023

Uh oh!

dgl-bot commented Sep 6, 2023

Uh oh!

dgl-bot commented Sep 6, 2023