Search Results on ListenNotes.com

Listen Notes is a search engine and podcast database. The technology is actually very boring. No AI, deep learning or blockchain. “If you must announce the implementation of AI, then you are not using Real AI” :)

After reading this article, you can repeat my project or easily do something similar. You won’t have to hire a lot of developers. Remember, when Instagram raised $ 57.5 million and went to Facebook for $ 1 billion , they had only 13 employees - and these are not only developers. Instagram purchase occurred in early 2012. Now is the year 2019, and today more than ever it is easy to create something meaningful with a tiny engineering team - even from one person.

If you have not seen Listen Notes, try it now:

Overview

Let's start with the requirements or features of the project.

Listen Notes provides two functions:

• Website for podcast listeners. There is a search engine, a database of podcasts, playlists for later listening , saved clips cut from podcasts , and notifications when the specified keyword is mentioned in some new podcast on the Internet.
  
• Search for podcasts and APIs for developers. We need to track the use of the API, receive money from subscribers, provide customer support and much more.

Everything works on AWS, only 20 servers in production (as of May 5, 2019):


Servers running Listen Notes

By host name, you can easily guess what each server does.

• production-web serves web traffic for ListenNotes.com.
  
• production-api serves the traffic API. We support two versions of the API (as of May 4, 2019): v1api (deprecated) and v2api (new)
  
• production-db launches PostgreSQL (master and slave)
  
• production-es launches the Elasticsearch cluster.
  
• production-worker runs stand-alone processing tasks to always keep the podcast database up to date and provide some magic features (e.g. ranking search results, recommendations for episodes / podcasts, etc.).
  
• production-lb is a load balancer. For convenience, I also run Redis and RabbitMQ on this server. I know that this is not perfect. But I'm not the perfect person. :)
  
• production-pangu is a production server on which I sometimes run one-time scripts and test changes. What is a pangu ?

Most servers can be scaled horizontally. That's why I call them production-something1 , production-something2 , etc. ... It’s very easy to add production-something3 and production-something4 to the cluster .

Backend

The entire backend is written in Django / Python3. The operating system is Ubuntu.

UWSGI is used to serve web traffic. I installed nginx before uWSGI processes, it also works as a load balancer.

The main data warehouse is PostgreSQL , with which I have extensive experience in development and operation for many years. With proven technology, you sleep peacefully at night. Redis is used for various purposes (e.g. caching, statistics ...). It is easy to guess that Elasticsearch is used somewhere . Yes, I use it to index podcasts and serve search queries, like most boring companies .

Celery is used for offline processing, and Celery Beat is intended for scheduling tasks that are similar to Cron tasks, but a little nicer. If in the future Listen Notes becomes popular, and Celery and Beat cause problems with scaling, I will probably switch to two projects that I did for the previous employer: ndkale and ndscheduler .

Supervisord manages the processes on each server.

Wait, what about Docker, Kubernetes, and serverless architecture? Nothing. With experience, you learn not to do too much. In fact, I worked a bit with Docker back in 2014 at a previous job: what was good for an average startup worth $ 1 billion seems unnecessary for a tiny one-person company.

Frontend

The web interface is mainly built using React + Redux + Webpack + ES . Pretty standard these days. When deployed in a production environment, JS packages are downloaded to Amazon S3 and issued through CloudFront .

On ListenNotes.com, most web pages are half rendered on the server side ( Django template ) and half on the client side ( React ). The web page template comes from the server, and on the client side, the interactive web application is mainly rendered. But several web pages are fully prepared on the server because of my laziness and some potential SEO benefits.

Audio player

I am using a highly modified version of react-media-player . It works on the website as a built-in player on Twitter and a built-in player on third-party sites:


Built-in player on third-party sites

API

We provide developers with a simple and reliable podcast API . Building an API is similar to building a website. Here's the same Django / Python stack for the backend and ReactJS for the interface (e.g. API toolbar, documentation ...).


Listen API Toolbar


Listen API Documentation

For the API, we need to track how many requests the client uses in the current billing cycle, and charge a fee. It is easy to imagine that Redis is actively used here :)

Devops

Machine Preparation and Code Deployment

Ansible is used for provisioning. Essentially, I wrote a bunch of yaml files to indicate which configuration files and what software should be on each type of server. I can deploy a server with all the correct configuration files and all installed software with the click of a button. Here is the directory structure for these Ansible yaml files:


I could better name the directories. But then again, that’s enough now

Ansible also helps to deploy code in production. Basically, I have a deploy.sh wrapper script that works on macOS:

  ./deploy.sh production HEAD web 

This script takes three arguments:

• Environment : production or staging.
  
• The version of the listennotes : HEAD repository means "just to deploy the latest version." If a SHA commit is specified, it will deploy a certain version of the code - this is especially useful when I need to roll back after a bad deployment.
  
• Type of servers : web, worker, API or all. I do not need to deploy to all servers at once. Sometimes I make changes to Javascript code, then I need to deploy it only on the web, without touching the API or the workers.

The deployment process is mainly organized by Ansible yaml files, and, of course, is extremely simple:

• On my Macbook Pro , if it's a deploy for web servers, then Javascript packages are created and uploaded to S3.
  
• On target servers, git clones the listennotes repository into a folder called timestamp, checks for a specific version, and installs new Python dependencies, if any.
  
• On the target servers, a symbolic link points to the aforementioned folder called timestamp, then we restart the servers using supervisorctl.

As you can see, I do not use these fancy CI tools. Only the simplest and most reliable tools that really work.

Monitoring and Alerting

Monitoring and alerts is done by Datadog . A simple dashboard shows some high-level metrics. Everything here is designed to increase my confidence when fussing with servers in production.


Datadog Dashboard for Listen Notes, as of December 2017

Datadog is connected to PagerDuty. If something goes wrong, PagerDuty will send me a notification by phone and SMS.

I also use Rollbar to track the status of Django code and catch unexpected exceptions, notifying me by email and Slack.

Slack is very active. Yes, this is a company with one person, so it is not needed for communication, but for monitoring interesting events at the application level. In addition to integrating Datadog and Rollbar with Slack, the Slack backend web code also integrates Slack incoming web hooks to notify you of user registration or some interesting actions (for example, adding or deleting elements). This is a very common practice in technology companies. If you read books about the early years of Amazon or PayPal, you will find out that both companies had a similar notification mechanism: whenever a user registered, a ding was heard to notify everyone in the office.

Since the launch in early 2017, Listen Notes has not had a serious downtime (more than 5 minutes), except for this . I am always very careful and practical in working DevOps. For servers, a serious overhead is provided in case of some huge surge in attendance due to getting into the press or something else.

Development

I work for WeWork in San Francisco. Some may ask, why not just work from home or from some casual cafes. Well, I really value productivity and am ready to invest money in it. I do not believe that a home mess contributes to the development of software (or any work in the field of knowledge / creativity). I rarely work more than 8 hours a day (sorry 996 people ). I want every minute counting. Thus, a good and relatively expensive private office is what I need. Instead of saving money to the detriment of time, I optimize time to use it profitably and make money.


My office at WeWork

I work on a MacBook Pro. An almost identical infrastructure runs inside Vagrant + VirtualBox . For the development environment, Vagrant uses the same set of Ansible yaml files described above.

I support the philosophy of a monolithic repository . Thus, there is one and only one listennotes repository with DevOps scripts, frontend and backend code. It is hosted as a private repository on GitHub. All development is in the main branch. I rarely use brunches.

I write code and run dev servers (Django runserver and webpack dev server) using PyCharm . Yes, I know, it's boring. After all, this is not Visual Studio Code or Atom or some kind of cool IDE. But for me, PyCharm works just fine. Old school, what can you do.


My pycharm

Miscellaneous

There are tons of useful tools and services that I use to create Listen Notes as a product and company:

• iTerm2 and tmux as terminals.
• Notion for TODO lists, wikis, notes, design notes ...
• G Suite for @ listennotes.com mail, calendar, and other Google services.
• MailChimp for monthly mailing .
• Amazon SES for business and some marketing newsletters.
• Gusto to pay yourself and some contractors if they are not with Upwork.
• Upwork for finding contractors.
• Google Ads Manager for managing ads and tracking performance.
• Carbon Ads and BuySellAds for backup ads.
• Cloudflare for managing DNS, CDN and firewall.
• Zapier and Trello to streamline workflows in the podcast interview section.
• Medium for a company blog.
• Godaddy and Namecheap for domain names.
• Stripe for receiving money from users (mainly for the API ).
• Google speech-to-text API for decoding podcasts.
• Kaiser Permanente for health insurance.
• Stripe Atlas for legal registration Listen Notes, Inc.
• Clerky : generating legal documents for venture investors (SAFE) and hiring contractors if they are not with Upwork.
• Quickbooks for bookkeeping.
• 1password to manage passwords in a heap of services
• Brex as a payment card - you can get additional $ 5,000 AWS credits using them on top of AWS credits from WeWork or Stripe Atlas.
• Bonvoy Business Amex Card - you can accumulate Marriott Bonvoy points on luxury hotels and flights. This is the best travel bonus program /
• Capital One Spark for your current bank account.

Keep calm and don’t jerk ...

As you can see, we live in a great time to launch our business. There are so many off-the-shelf tools and services that save time and money, increasing your productivity. Now is the best time in history to create something useful for humanity with a tiny team (or only with the efforts of one person), using simple and boring technologies.

Over time, companies become smaller and smaller. No longer need to hire tons of full-time employees. To do all the work, you can use SaaS services and contractors to order.

Basically, the main obstacle in creating a project is excessive thinking. What if that, what if that. Boy, nobody needs you. Everyone is busy with their own affairs. Nobody is interested in you and your project until you prove that you are worthy of attention. Even if you lock the launch, no one will notice. Think big, start small, act fast. It’s absolutely normal to use boring technology and start with something simple (even ugly) if you really solve the problem.


Now there are so many people with a cargo cult . Ignore the noise. Keep calm and don't twitch.