Skip to main content

Build A MVP: SpyGather.com - How I built it

· 8 min read
Iain Cambridge

I'm planning on starting a monthly blog series where I build an MVP to solve a specific problem I'm having and want a tool to solve and share how I built it. My thinking is, one, it'll be useful for people to see how to build an MVP, two, it'll be useful for people to see what Parthenon can build, and three, it's just kind of a cool, unique idea I think would be fun. If you like the idea of monthly MVP builds detailed out, sign up for the newsletter.

Problem​

The first thing is to find a problem to solve. In this case, it was that I wanted to get notifications of threads and comments on Reddit about specific keywords. The first kind of obvious reason is to find mentions of Parthenon so I can reply. Then it's to find conversations about SaaS Bootstraps so I can join those either to learn more about what they want or to help explain the subject. Then there are threads where they're semi-related to what Parthenon can provide, but the question is super basic and common, so they end up getting moderated really quickly. I think messaging these people and helping them anyways might be helpful.

Start the build​

So the first thing is clear, I create a new skeleton application with Parthenon. Configure the basic things such as email configuration. For this build, I don't want to start having team options as I want this to be a Micro-SaaS that solves a very specific problem at a low price. But... I do want the ability, later on, to expand and allow people to have teams. So to start off with, I've left the teams functionality enabled and just removed the team's page in the application dashboard. So while the system underneath it has teams support, the user doesn't know about that ability.

Now I have a basic skeleton for the application and I just need to start adding the logic to solve my problem.

Scraping Reddit content​

Since this is an MVP and doesn't need to worry about full-scale issues. My initial approach is to build Symfony commands that can be run every minute. This will give me notifications in a reasonably quick speed. Another decision I made due to this being an MVP that just needs to solve my problem in a viable manner is that I choose which subreddits I want to search. For example, if you're mentioning Parthenon on the Athens subreddit there is an extremely good chance you're not even remotely interested in my Parthenon. This meant I didn't need to worry about handling the full wave of data Reddit generates and could just deal with the few specific subreddits that I have created searches for. There are also two different scrape commands, one for threads and one for comments.

The Thread Scrapper​

The process for the thread scrapper is rather simple.

  • It fetches all the subreddits that exist in the database. A subreddit only exists in the SpyGather database if a search has been created for it.
  • Once it has fetched the subreddits it loops through them and fetches the newest threads for that subreddit.
  • It loops through the threads and sees if there is already an entry in the database for this thread
  • If so it fetches all the data from the databases
  • If not it saves it to the database
  • It fetches the searches that are valid for that subreddit.
  • It loops through the searches and sees if they are a match for the thread
  • If there is a match, it checks to see if there has already been an alert for this search
  • If there has been no alert, it creates an Alert.

The Comment Scrapper​

The process for the comment scraper is quite similar to the thread scrapper. Since this is an MVP, I decided to only care about active threads, so ones that have been created within the last 72 hours.

  • It fetches all the threads that were created within the last 72 hours and haven't been checked in 5 minutes.
  • It fetches the data for the thread
  • It loop through the comments
  • It checks to see if the comment is in the database
  • If it is it reads the data from the database
  • If not it saves data to database
  • It fetches the searches for the subreddit the thread is in
  • It loops through the searches and sees if they are a match for the comment
  • If there is a match it checks to see if there has already been an alert for this search
  • If there has been no alert it creates an Alert.

Matching And Alerts​

Alerts are created when there is a match. Since this is a MVP, the matching is super simple. It just does a regex case insensitive search. It is possible to make a much more complex and powerful matching system that allows for finding positive or negative comments. However, this is just an MVP and being able to just find threads and comments that have the keyword does enough to solve my problem.

The MVP has two ways of getting alerts. Email and Slack. If this was just a standard MVP, I would have just added Emails, but since this is something I want to use, I really like my notifications in Slack. So I added slack.

With the email sending, I just reused the Parthenon email sending using templates with Sendgrid and populated the data with the data from the thread and comment if there was a comment.

With the slack notifications, I reused the slack webhook message sending and the slack message building functionality within Parthenon.

I also wanted the ability to view the alerts within the application, so I built a UI that listed out the matches and showed me all the data for the post, comment, and search all together. So far, I don't really use this that much as all the information I need to take action on the alert is in the notification. So this was kind of just really for show and historical purposes.

I realised once this was built that I've had the notification requirement quite a few times. So I added a notification control ui to the Parthenon roadmap, which is the next feature to be added and hopefully will be available real soon.

Adding Plans And Payments​

Since I want to allow other people to use this system, I wanted to have plans so there is a free plan and then a paid plan. Payments are handled via Stripe as that is what Parthenon supports. So I created prices for the paid plan, a yearly and a monthly price. I added the Stripe public and private API keys, as well as the price ids to the Parthenon config.

Here I just used the plans support that is in Parthenon. This is defining the limits, which I decided was on the number of searches you could have and how many alerts you got per week.

And then the UI just worked as the skeleton application comes with a plan page.

Landing Page and Launch​

I added an extremely rough landing page that has the features that are available. Normally, I hire a designer to create landing pages and often I have a copywriter write the copy. This time I just wanted something that I could work on and improve later. I've noticed that lots of products have rather basic landing pages but still convert. But also, for me, selling isn't the biggest point here, it's that it solved my problem.

I started the project on the Friday and I had a usable MVP deployed by Monday. I would say I spent about 20 hours in development time over the weekend to build this MVP. You can check it out at spygather.com)

If you're interested in more of these kinds of posts you can subscribe to the newsletter. You can follow me on Twitter at TheSaaSDev.

And a blog post to promote Parthenon wouldn't be same without a call to action. If you've read my last blog post, you'll realise I just gone full time and created a lifetime 50% off voucher IGOTFIRED which is only valid until the 1st of May. Also if you want to find out the absolutely crazy story of my firing once the legal case is over subscribe to the newsletter. It's completely nuts!