domain

youtube-dl and embedded Vimeo

 in Applications  tagged domain / referer / vimeo / youtube-dl by

Youtube-DL being an amazing tool as it is, has a way to download Vimeo videos that give the following error:

Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.

To do so, we needed to use the --referer parameter with the value of the website that we found the embedded video on. For example:

youtube-dl -v "https://player.vimeo.com/video/622fa342f" --referer "https://example.com/courses/intro/hello/";

The --referer parameter specifies a custom referer and can be used if the video access is restricted to one domain.

(youtube-dl) tux@bob:~$ youtube-dl -v "https://player.vimeo.com/video/622fa342f"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://player.vimeo.com/video/622fa342f']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.7 (CPython) - Linux-5.13.0-44-generic-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg present, ffprobe present
[debug] Proxy map: {}
 622fa342f: Downloading webpage
ERROR: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 636, in _real_extract
    webpage, urlh = self._download_webpage_handle(
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 652, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/tux/anaconda3/envs/youtube-dl/lib/python3.9/site-packages/youtube_dl/extractor/vimeo.py", line 643, in _real_extract
    raise ExtractorError(
youtube_dl.utils.ExtractorError: Cannot download embed-only video without embedding URL. Please call youtube-dl with the URL of the page that embeds this video.

Cannot verify domain with Yandex when domain is behind CloudFlare 2

 in HowTos  tagged cloudflare / cname / domain / yandex by

Recently we were trying to verify the ownership of a domain through yandex. We tried the CNAME approach which would be more universal and so we added a new CNAME record in the DNS configuration in CloudFlare.

The record had the following configuration:

  • Type: CNAME
  • Name: yamail-dd63c3831dbd
  • Value: mail.yandex.com
  • TTL: Automatic
  • Status: DNS and HTTP proxy (CDN)

We tried several times the verify domain button in https://domain.yandex.com/domain/example.com/ but it kept on failing saying that the CNAME record was not found. Only after we disabled the DNS and HTTP proxy (CDN) did it work.

So in the end, the properly working record was as follows:

  • Type: CNAME
  • Name: yamail-dd63c3831dbd
  • Value: mail.yandex.com
  • TTL: Automatic
  • Status: DNS Only