ShopMWBot
The web crawler operated by Shop.mw for indexing Malawian content.
If you found ShopMWBot in your server logs, this page explains what it is, why it
visited your site, and how to make it stop if you don't want your content indexed.
What is ShopMWBot?
ShopMWBot is an automated web crawler operated by Shop.mw, a Malawian search engine and discovery platform. It is owned and managed by JADE IT, a registered company in Blantyre, Malawi.
The crawler visits publicly accessible web pages to build Shop.mw's search index. Content indexed by the crawler may appear in search results on shop.mw, with attribution and a link back to the original source.
Crawler Identity
What the Crawler Does
- Visits publicly accessible pages — pages anyone can see without logging in.
- Extracts visible text — page titles, descriptions, headings, body content, and images.
- Follows links — finds other pages on your site (and related Malawi sites) by following links in the HTML.
- Respects robots.txt — if your robots.txt tells ShopMWBot not to crawl, it won't.
- Identifies itself — always sends the User-Agent string above, so you can spot it in logs.
- Rate-limits itself — one concurrent connection per domain with a 1-second delay between requests.
- Respects your submissions — if you or a user of Shop.mw submits your site through a profile, we crawl it at your chosen frequency, or stop at any time on request.
What the Crawler Does NOT Do
- Does not bypass authentication — if a page requires login, it's not crawled.
- Does not pay or bypass paywalls — content behind paywalls is not indexed. If a page returns a login prompt instead of content, we recognize that and skip the page rather than indexing the prompt text.
- Does not execute JavaScript — the crawler sees the HTML source, not the rendered page. If your content is loaded via JavaScript, the crawler likely won't see it.
- Does not scrape email addresses — no harvesting from mailto: links or page content.
- Does not crawl social media — Facebook, Instagram, TikTok, and similar platforms are explicitly excluded.
- Does not index private data — no private messages, no drafts, no unpublished content.
- Does not sell your content — indexed content is displayed only in Shop.mw search results, always linking back to your site.
- Does not republish full articles — search results show only a short snippet and a link back to your site.
Why Your Site Was Visited
ShopMWBot likely found your site through one of these paths:
- A seed list — we maintain a curated list of Malawian websites to crawl
- A link from another site — a page we already indexed links to your site
- A user submission — a Shop.mw user submitted your website or RSS feed through their profile
- A domain ending in .mw — we prioritize Malawian domains (.mw, .co.mw, .ac.mw, .gov.mw, .org.mw, .net.mw, .edu.mw, .com.mw)
Submission Through Profiles
Website owners and users can submit sites for crawling
Any registered Shop.mw user can submit a website through their profile page under Profile → Submit your website for indexing. Website owners can use this to make sure their content is discovered without waiting for the crawler to find it organically.
When submitting, the user chooses a crawl frequency:
- Every 6 hours
- Every 12 hours
- Daily
- Weekly (default)
- Monthly
Submitted sites are added to our crawl queue at priority level 7 — below our curated Malawi news and government sources, but above weekly newsletters. This ensures user-contributed content is indexed without overwhelming the crawler.
Submissions can be removed by the user at any time, which stops future crawls of that site.
What Appears in Search Results
When your site is indexed, Shop.mw may display:
- The page title
- A short snippet of visible text (usually the first 100–500 characters)
- The page URL, linking back to your site
- An image from the page (if one is present)
- The publication date (if the page provides one)
We do not display full articles. Users who click a result are sent directly to your site. Indexing is intended to drive traffic to your site, not replace it.
Crawl Frequency
How often ShopMWBot visits a site depends on the type of content it publishes:
| Site type | Typical crawl frequency |
|---|---|
| News sites | Every 12 hours |
| Government, education, and NGO sites | Daily to every 2 days |
| Business, blogs, and music | Weekly |
| User-submitted sites | Based on the frequency chosen at submission time |
If you want to request a lower or higher crawl frequency for your site, contact us at jadegraphicsmw@gmail.com.
How to Opt Out
You can opt out at any time
If you own or manage a website and do not want your content indexed by Shop.mw, you have two options.
Option 1 — Block the crawler with robots.txt
Add these lines to your robots.txt file (usually at
https://your-site.com/robots.txt):
User-agent: ShopMWBot Disallow: /
ShopMWBot honors this directive and will not crawl any part of your site going forward.
Option 2 — Request removal by email
If you want existing indexed content removed from Shop.mw's search results, send an email to:
Please include:
- The domain name or specific URLs you want removed
- A statement confirming you own or manage the site
- Your name and contact information
We process opt-out requests within 7 working days. Once processed:
- Existing indexed content is removed from search results
- Your domain is added to our blocklist
- Future crawls will skip your site entirely
Rate Limiting and Server Load
ShopMWBot is designed to be gentle on servers:
- A minimum delay between requests to the same site
- Only one concurrent connection at a time per domain
- A maximum number of pages per crawl session
- Automatic backing off if the server returns errors
- Automatic skipping of pages that return "access denied" or login-required responses
If you believe ShopMWBot is still causing performance issues on your server, contact us at jadegraphicsmw@gmail.com with details (timestamp, IP address, request pattern) and we'll investigate immediately.
Verifying ShopMWBot
To confirm a request in your logs came from us:
- The User-Agent header should contain
ShopMWBot/1.0 - The request will reference
https://shop.mw/botin the UA string - If you're not sure, forward the log entry to jadegraphicsmw@gmail.com and we'll confirm or deny it
If you see requests claiming to be ShopMWBot from a source that seems suspicious, please report them — someone may be impersonating the crawler.
Frequently Asked Questions
No. The crawler is deliberately throttled. If you notice a problem, contact us and we'll reduce or stop crawling your domain immediately.
No. If your site is public and linked from other Malawi sites, it will likely be discovered automatically. To guarantee discovery, you can create a Shop.mw account and submit your URL through your profile, or email us and we'll add it as a seed.
Yes. Email us the URL and we'll re-crawl it within the next cycle. Fresh content is normally picked up automatically on the next crawl.
No. Search results show only a short snippet (typically 1–2 sentences) and a link back to your site. We do not republish full articles. This is consistent with standard search engine practice and fair use.
No. Social media platforms are explicitly excluded from crawling.
No. Pages that require a login or paid subscription are not indexed. If your site returns a login prompt to our crawler, we recognize it and skip the page. We respect the choice of publishers to keep their content behind a paywall.
It depends on the site type. News sites are visited every 12 hours, government and education sites daily, and most other sites weekly. Sites submitted through a profile are crawled at the frequency chosen by the submitter. You can request a lower or higher frequency at any time.
Yes. Search for site:your-domain.mw (or your full domain) on
shop.mw/search to see which pages from your site
appear in our index. If something is missing or incorrect, email us and we'll fix it.
Email jadegraphicsmw@gmail.com with the domain, the URL, and a description of the issue (log entries are especially helpful). We respond to website-owner inquiries within a few working days.