Day 9 started the fun of “to solve this problem, the logic is easy, but since we need continual state, it’s going to take a long time” advent of code solutions. My sample pipeline solution took 1 minute to run, and that was for a total of under 100 “head” movement. Then I saw the input file had 2,000 lines and realized this was going to be an over 1-hour process to get completed. My first iteration was able to process 1.7 documents per second (where each document is a 1-unit step in the puzzle) which my input file had 11,240 steps, so that took roughly 2 hours to complete (8 minutes short of 2 hours). I’m currently running with the setup you see in this post, I assume it will work properly, but I’m currently processing at 2.4-2.5 documents per second. Based on the new speed, it SHOULD complete in about 80 minutes. This is still a long time, but I’m still impressed that this is even possible in SOMEWHAT of a normal timeframe.
The improvement in speed is that rather than having the tail movement pipeline read from the file and write to the file, I’m simply having it update the data in-place, with a pipeline parameter to define the index of the output array to actually process. This was previously done via passed-in parameter and file name for the index and filename where it was read, and then all processing done. That Disk IO really does impact the processing time for things, so these are the scenarios where it’d be nice to have something to do with state, etc. I would say that today’s puzzle is more of a practice in patience and really testing every step of a sub-pipeline that you may need to process. While testing is hard, you can see that I’ve just leveraged a JSON Generator that I disable and disconnect for final running for the internal values. I also just disable the pipeline calls to get the first state of things configured to begin with.
Another interesting problem, and unfortunately a very long runtime to get the solution. Below are the screenshots from the runtime of the pipelines, then the pipelines themselves and SLP Files (I’ll edit/update to add the second runtime time screenshot)
With sub-sub-pipeline reading/writing file (1 hour, 51 minutes):
With sub-sub-pipeline just editing the structure itself (1 hour, 18 minutes):
Screenshots:
SLP Files:
day09_2022_12_09.slp (32.4 KB)
day09_move_2022_12_09.slp (38.3 KB)
day09_tail_move_2022_12_09.slp (10.7 KB)
Day 10 was a rather interesting problem to do via SnapLogic, and I thought I might already have to deal with more state-saving in a loop, but I found a way around it this time (yes, I’m very excited about this). We know this input is pretty small based on part 2, but by collecting all inputs, you can calculate the sum of some value at a given index with slice and then reduce for that value. The hardest part for this was determining what each part was really wanting when it comes to those values (I struggled with that when I solved the problem via Java at midnight).
Frankly, there’s nothing super stand-out of this solution, other than again finding a way to potentially re-imagine the problem in a different frame. We could’ve done what we did yesterday for Day 9 and saved state in a loop, but with the smaller input data that we had, I figured this was a pretty good way to show how you can somewhat avoid having to save state in these kinds of situations.
Screenshot:
SLP File:
day10_2022_12_10.slp (38.1 KB)
Day 11 is one of those days/puzzles where SnapLogic simply makes things take a long time, but there were some shortcuts that I could and did take to speed it up. Every time you have to preserve state, you effectively incur a time penalty (this is why the script snap can be so handy in situations like these). My current solution for part 2 still hasn’t completed, but I’m confident (when comparing to my code-based one) that it will work without any issues. Currently, part 2 (which requires 10,000 iterations) is running at 1.2 rounds per second, so that SHOULD finish in 2 hours, 19 minutes (roughly) which seems to be yet another repeat of Day 9 with respect to speediness. The key with speeding up these iterations, again, is using plex-local files (file:///tmp/file).
For this solution, I have 5 total pipelines, one for the full day (day11), then 1 for each major part (part 1 and part 2) as I found some performance impact by using a pipeline parameter to determine further downstream operations and flip the inner per-monkey processing loop. While the two inner processing loops are similar, they weren’t similar enough to really be able to use a pipeline parameter (in my opinion) to deal with it, though I did work with the example (4 monkeys) vs. input (8 monkeys) input in a fairly good way (in my opinion). You’ll notice that my “round” pipelines have 8 pipeline executes in a row, which all layer on top of each other, since I know at most we’d have 8 monkeys to process, I figured repeated pipeline executes made more sense in the end than trying to do another loop with saving state. I already knew that 10,000 iterations would be rough to do saving state for each of those, but multiplying that by 8 would slow it down even more, so in this case, taking a manual step can really help (if you know the scope of limitations. This is also why there’s a router and union for instances where the inner array doesn’t match the number being processed, it short-circuits the processing a bit more.
At midnight when I finished my coded solution, I never thought I’d be able to finish this puzzle using SnapLogic, and it’s a LONG runtime, but I’m still pretty impressed with being able to write up the solution and execute it in some amount of time that might be considered “valid” in some definition. You’ll notice the part 1 and part 2 specific items LOOK the same, and they’re both very close, but it’s the inner pieces that are different, I probably could combine those into just 2 pipelines rather than 4, but figured keeping down on the potential introduction of more issues was the best way to go forward, I’ll make a few more updates (after it runs successfully) and report back as necessary, but at least the outer loop should be able to be generic to both parts without too many issues.
Screenshots:
SLP Files:
day11_2022_12_11.slp (55.5 KB)
day11_round_part1_2022_12_11.slp (28.3 KB)
day11_process_monkey_part1_2022_12_11.slp (31.9 KB)
day11_round_part2_2022_12_11.slp (28.5 KB)
day11_process_monkey_part2_2022_12_11.slp (32.3 KB)
