`scrobbleIngest` #13

New issue

Closed

opened 2025-06-12 00:09:43 +00:00 by mitteneer · 0 comments

mitteneer commented

2025-06-12 00:09:43 +00:00

Owner

See 36873053bc

Currently writing a function that takes in the data, and parses it, effectively writing my own jsonParse, but it's doing more than that.

Timeline:

Load in data, loop over all data to verify and turn into intermediate struct, loop again to add to job data, then run job on all data at once (run checks on each piece of data in job)
Load in data, loop over data to verify, turn into intermediate struct AND add to job data, then run job on all data at once (run checks on each piece of data in job)
Load in data, loop over data to verify, turn into intermediate struct, AND add to job data, then run job on all data at once (use hashmap before job to reduce number of checks in job)

Now, what I'm currently writing: Load in data as the intermediate struct, loop over data to verify and add to job data, then run job on all data at once (with hashmap)

What I intend to arrive at: Verify data as it's being read, check against hashmap to see if job needs to be run on that data, and then run individual jobs on each piece of data

See https://git.sawebb.xyz/mitteneer/zuletzt/commit/36873053bca53dc5d39f3d0578701805c966f4a9 Currently writing a function that takes in the data, and parses it, effectively writing my own `jsonParse`, but it's doing more than that. Timeline: 1. Load in data, loop over all data to verify and turn into intermediate struct, loop again to add to job data, then run job on all data at once (run checks on each piece of data in job) 2. Load in data, loop over data to verify, turn into intermediate struct AND add to job data, then run job on all data at once (run checks on each piece of data in job) 3. Load in data, loop over data to verify, turn into intermediate struct, AND add to job data, then run job on all data at once (use hashmap before job to reduce number of checks in job) Now, what I'm currently writing: Load in data as the intermediate struct, loop over data to verify and add to job data, then run job on all data at once (with hashmap) What I intend to arrive at: Verify data as it's being read, check against hashmap to see if job needs to be run on that data, and then run individual jobs on each piece of data