Catalog App Maintenance

After photos are sorted, they need to be re-indexed. Re-indexing converts each training photo into a unique numerical “fingerprint” that represents its features, then stores that fingerprint in the index with a label of which item that fingerprint belongs to. This is what lets the AI recognize items more accurately. The more "fingerprint" reccords an item has that the model can compare against, the more likely it will be to get an accurate match.

Make sure you have sorted all of the training photos you can before reindexing them. Only photos that have been sorted will be included in the re-index.

Here are the steps to re-index the sorted training photos:

1. Open the Cloudflare Worker page (link below)

   Open CF Worker Page

2. With the Cloudflare Worker tab open press Ctrl + Shift + I all at the same time to open Developer Tools

3. At the top of Developer Tools Window, there should be different icons for the different tabs. Click the Console tab. If you hover over the icons at the top with your mouse, it should tell you what each icon is for.

4. Copy the code below, paste it into the Console, and press Enter:

   Copy Code

   const BASE  = "https://acs-drive-proxy.acsformyard.workers.dev";
   const TOKEN = "refreshnow";
   
   await fetch(`${BASE}/recognition/admin/reindex/refresh-folders`, {
   method: "POST",
   headers: { "x-admin-token": TOKEN }
   }).then(r => r.json()).then(console.log);

   This code resets the training counter to 0 for a fresh training run.

5. Read what that command returned. It should say something like this:

   {ok: true, totalFolders: ###}

   The ### will be the number of folders in the Google Drive's Recognition > Items folder.

6. If everything looked good in Step 5, copy the code below and paste it into the Console. Check the number of maxBatches in the code and make sure that it matches the number of folders in Step 5. To change it, edit it after pasting to the browser console just like a text document.

   Once you've made sure that matches up, press Enter to run the worker script.

   Copy Code

   const BASE  = "https://acs-drive-proxy.acsformyard.workers.dev";
   const TOKEN = "refreshnow";
   
   let limitFolders = 1;
   let maxChanged   = 12;
   let pauseMs      = 2000;
   const maxBatches = 520; // This number should be equal to the totalFolders number from the last step.
   
   const sleep = (ms) => new Promise(r => setTimeout(r, ms));
   
   (async () => {
     for (let i=1;i<=maxBatches;i++){
       const url = `${BASE}/recognition/admin/reindex/run?limitFolders=${limitFolders}&maxChanged=${maxChanged}`;
       let resp, json;
       try {
         resp = await fetch(url, { method: "POST", headers: { "x-admin-token": TOKEN }});
         json = await resp.json();
       } catch (e) {
         console.warn(`#${i} network`, e);
         await sleep(pauseMs); continue;
       }
   
       if (json?.ok) {
         console.log(`#${i} OK`, { limitFolders, maxChanged, counts: json.counts, cursor: json.cursor });
         await sleep(pauseMs); continue;
       }
   
       const err = (json?.error || "").toLowerCase();
       if (err.includes("subrequest") || err.includes("cpu") || resp?.status === 503) {
         // back off
         if (maxChanged > 8) maxChanged = Math.max(8, Math.floor(maxChanged/2));
         else if (limitFolders > 1) limitFolders--;
         pauseMs = Math.min(15000, pauseMs + 2000);
         console.warn(`#${i} backoff`, { limitFolders, maxChanged, pauseMs, err: json?.error });
         await sleep(pauseMs);
         continue;
       }
   
       console.warn(`#${i} other`, json);
       await sleep(pauseMs + 3000);
     }
   
     // final status
     const s = await fetch(`${BASE}/recognition/admin/reindex/status`, { headers: { "x-admin-token": TOKEN }});
     console.log("STATUS", await s.json());
   })();

7. Wait for the training run to finish. You can work on other things while this is processing, but do not close the tab or window — if you close it, the run will stop.

   While the training run is happening, the console will return different status reports that look something like this:

   #1 OK ▶ {limitFolders: 1, maxChanged: 12, counts: {…}, cursor: 1}

   VM60:24

   #2 OK ▶ {limitFolders: 1, maxChanged: 12, counts: {…}, cursor: 2}

   VM60:24

   #3 OK ▶ {limitFolders: 1, maxChanged: 12, counts: {…}, cursor: 3}

   VM60:24

   #4 OK ▶ {limitFolders: 1, maxChanged: 12, counts: {…}, cursor: 4}

   VM60:24

   …(new lines will appear every few seconds)…

   New lines will continue popping up until it has ran through the same number of lines as the totalFolders and maxBatches.

   After each folder has been processed it will give a final STATUS update. When it has finished running, it should look like this:

   #519 OK ▶ {limitFolders: 1, maxChanged: 12, counts: {…}, cursor: 519}

   VM60:24

   STATUS ▶ {ok: true, status: {…}}

   VM60:24

8. If the STATUS is {ok: true, status: {…}}, then the training run has been successfully completed!

If a certain item doesn't have any training photos for the AI to learn from or compare to, then that item won't show up in the results at all for the Search By Photo feature of the app. If an item has a small number of training photos for the AI to go off of, then the results are less likely to be accurate. The more photos a given item has, the more likely you are to get a good result from the Search By Photo feature.

Manually combing through the folders would take forever, so we have a custom-built tool that we can use to search through those folders for you and download a CSV file that you can open in excel to view and sort the results. That will help to get a count of how many sorted photos there are for each item, and will help to fill in gaps in the training data.

(This can be run at any point, regardless of when the last training run happened)

Here is how to run it:

1. Open the Cloudflare Worker page (link below)

   Open CF Worker Page

2. With the Cloudflare Worker tab open press Ctrl + Shift + I all at the same time to open Developer Tools

3. At either the top or in a sidebar of the Developer Tools Window, there should be different tabs you can swap between. Click the Console tab. In Microsoft Edge, you can hover over the icons in the sidebar with your mouse to show what tab each icon is for.

   If there is a 404 error in there, just refresh the page and that should get rid of it.

4. Copy the code below, paste it into the Console, and press Enter:

   Copy Code

   const BASE  = "https://acs-drive-proxy.acsformyard.workers.dev";
   const TOKEN = "refreshnow";
   
   await fetch(`${BASE}/recognition/admin/reindex/refresh-folders`, {
   method: "POST",
   headers: { "x-admin-token": TOKEN }
   }).then(r => r.json()).then(console.log);

   This code resets any folder increment counters to 0 and counts the total number of folder.

5. Read what that command returned. It should say something like this:

   ▶ {ok: true, totalFolders: ###}

   The ### will be the number of folders in the Google Drive's Recognition > Items folder.

6. If everything looked good in Step 5, copy the code below and paste it into the Console. Check the number of maxFolders in the code and make sure that it matches the number of folders in Step 5. To change it, edit it after pasting to the browser console just like a text document.

   Once you've made sure that matches up, press Enter to run the worker script.

   Copy Code

   const BASE  = "https://acs-drive-proxy.acsformyard.workers.dev";
   const TOKEN = "refreshnow";
   
   const pageSize = 1;
   const maxFolders = 520;  // This number should be equal to the totalFolders number from the last step.
   const pauseMs = 500;
   
   const sleep = (ms) => new Promise(res => setTimeout(res, ms));
   
   async function fetchTrainingCsvInChunks() {
     let start = 0;
     let csvParts = [];
     while (start < maxFolders) {
       const url = `${BASE}/recognition/admin/training-csv?start=${start}&limit=${pageSize}`;
       const resp = await fetch(url, { headers: { "x-admin-token": TOKEN } });
       const text = await resp.text();
       csvParts.push(text);
   
       const next = resp.headers.get("x-training-next-start");
       if (!next) break;
       start = Number(next);
       if (start >= maxFolders) break;
       await sleep(pauseMs);
     }
     const csv = csvParts.join(""); // combined CSV covering all fetched folders
     console.log("Combined CSV length:", csv.length);
     return csv;
   }
   
   fetchTrainingCsvInChunks().then(csv => {
     const blob = new Blob([csv], { type: "text/csv" });
     const url = URL.createObjectURL(blob);
     Object.assign(document.createElement("a"), { href: url, download: "training_folders.csv" }).click();
     URL.revokeObjectURL(url);
   });

7. Wait for the Cloudflare worker to sort through all of the folders. You can work on other things while this is processing, but do not close the tab or window — if you close it, the worker will cancel the action.

   The Cloudflare Worker will return a pending response like this to indicate that it is in the process of counting through the photos in the folders:

   ▶ Promise {<pending>}

8. When the worker is done sorting through the photos, it will automatically compile the results and download them as a file named training_folders.csv. This file can be opened in Microsoft Excel like a spreadsheet.

9. (Optional) To make it easier to sort and read the data once you have opened the file, you can select all of the columns and rows with data from the csv and select "Format as Table" in Excel's Home tab.

   Make sure to check the "My table has headers" box when creating the table.

   If you do this optional step, you'll be able to sort by item number and filter based on Forming System. You will also be able to filter the items based on the number of photos they have.