Ok, so my team member and myself both ran the exact SAME pipeline and the filter worked when she ran it to filter a huge dataset for one student, but whenever I tried it, the filter did not work. It’s just a simple: $Student_ID = “123” type filter.
So, I finally cut the one student and put him/her into a separate json file near the top of the json file (the second student from the top). THEN the filter worked for me. There’s over 6 million rows in this json file, which is why I started thinking about the size of the file and the fact that this particular student started on row 703,250.
Also, WITHOUT the modification above, if instead of filtering I write to a csv, the student also makes it into the csv!
The ONLY difference between my pc and her pc is that she’s on Windows 10 and I’m on Windows 11.
What could be causing this issue?
