Omnindex Journal 25/mar - Lots of updates
Last week I got Corona, so while resting at home I had plenty of time to work on Omnindex (in the periods I was not agonizing), but I didn't quite feel like writing about it, so now I have quite a lot of accumulated updates, most of it not super interesting tech, just usual web development, let's go:
*checks git log*
1. I have finalized the extension to sync all bookmarks:
The most tricky part was on how to connect the extension with your Omnindex account, what I do is basically generate a token for the browser, which is passed via browser events, using a bridge javascript that is injected in the page, but only if the domain is omnindex.com, which then connects back to the extension. Then I use the user agent to identify which browser you are using for a nice UI, but if you already have a similar browser connected, I offer the possibility to reuse the same token:
This is to solve the issue if you have 2 devices with the same browser, with bookmarks synced in between then, you don't need to duplicate all indexing in Omnindex side as well
Another tricky part is to get the extension to work in both firefox and chrome. This used to be a no-brainer with the WebExtension standard, but then chrome decided to move on to Manifest v3, and firefox can't do nothing except follow it, but it hasn't still. So instead of background scripts, chrome now requires us to use WebWorkers which reacts to events. Fortunately, my whole extension is only that, reacting to bookmark events, so I basically didn't have to change the code, but I do have to hold two versions of manifest.json, depending on which browser I'm building to now.
2. I have greatly improved the providers page, where you set up your connections
And the provider details, you can now see a list of everything that was saved already, and when where they synced. I for instance have 386 pages of indexed twitter likes
I plan to maybe later add the possibility to have better control of those, so you can block what you don't want to be on the search for example, or maybe tag them and organize it somehow, if you are into that
3. I've also improved the sign in and sign up pages and the onboarding flow, so that right after signing up the website guides you in setting up your providers, so you have *something* to search and actually experiment the service
We suggest some initial providers for you to setup, and after you are done you can go and search the things you indexed
…except because it's probably a lot of data and because of semantic search using the deep learning model to encode each sentence, it actually takes a while to index everything, so I added this piece of UI to tell the user their items are not ready yet, and that's why some of it will show up on the search results just yet
I really want users to have a great onboarding experience, and really understand the flow of the product and how it can be useful for them, I hope those provide good help!
4. I configured Sentry to capture and report exceptions, so I can move fast and break things
5. I've updated the top bar to look similar both when you are on search and configuration pages, to feel like you are still in the same product, and search anytime. Before I had a different topbar for the internal pages, so it looked like you were on a different app, now the only thing is that I have a black background on it when you are in internal vs search page
This was interestingly much harder to do in CSS then you might think, because of responsiveness. First because of the alignment we have on the search, if on a big screen, where the results are aligned with the search bar:
While on a smaller screen it shifts left:
And on mobile the search bar jumps from the middle to below the logo and menu button:
I made some smart use of CSS grid (first time I used it, and it fit like a glove to the problem!) and some workarounds to paint it black but only the header for the internal pages
Oh, btw, I also improved the account menu:
Before they were just plain links on the top bar
6. I've added a proxy to twitter avatar images
So, earlier I was just storing the twitter avatar url, and using it directly on the img tag, turns out, first of all, those images are not always available, returning 404 after a while. This is because twitter expires the old avatar if the user ever changes it to a new one, so we need to query the API again to retrieve a fresh url. Secondly, my firefox was blocking those images, because of risk of tracking, and very right to do so, indeed I was leaking data about my users from Omnindex to twitter.
Then I created a /twitter-img-proxy route, with a controller that proxies the request to twitter, and if it's a 404, it actually calls the API to retrieve a new image, update it in the database and returning the new one instead of a 404
In addition to all this, I've spent quite some time fixing bugs here and there, and still have a lot of them to fix, but feel like moving forward a lot :)
See ya!