File size limit

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Thu Jun 10, 2021 9:18 pm

BlompTeam wrote:
Thu Jun 10, 2021 6:11 pm
...I'll ask someone to upload a large (over 5G) file to Blomp via Rclone and save the logs for you. Rclone supports this natively with Swift. The manifest file is just created at the end of the upload usually.
Forgot to reply to that part. Thanks for asking. Do you have a ref. settings for chunker as you say it works? Like md5 is supported by blomp and sha1 is not, does it use the same format as manifest file created by rclone? Here is example:

rclone copy 10G.img remote:testimages . If I copy it that way with rclone chunker, then manifest file will be in same folder as chunked files and as far as I can see, blomp uses /filename/segement/##### where manifest file is in /. Beside that, rclone wants to have filename.# format, default is filename.rclone_chunk.###. By that here would be my second wish, if possible, it would be great if rclone chunked files appear as one file. They do appear as one file if I use rclone or simply mount chunker remote. If/when your chunking on your backend works again, this option will not be required. Also, if you some enable that *.rclone_chunk.### appear as one file if there is a manifest file. Rclone chunker uses by default rename, which is painfully slow for blomp due to copy/delete operation, using no rename leaves additional suffix to chunked files but would be as fast as uploading without chunker overlay as files do not need to be moved.

I truly believe it worked sometime in past when you tested it but I can assure you that it does not work since I registered.

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Fri Jun 11, 2021 5:10 pm

@BlompSupport
and here you go, this is what drives everybody mad and why NOBODY SHOULD USE BROWSER to upload files.

I am waiting for 45G file to be uploaded, I see 34 segments, however, it is all in wrong folder and as your service is not capable of moving files, instead reupload has to be done. This is just disappointing to the level to turn a back and never come back, as this happens permanently and your support ignores it despite many reporting about it.

BlompTechie
Posts: 16
Joined: Thu Feb 25, 2021 1:01 pm

Re: File size limit

Post by BlompTechie » Mon Jun 14, 2021 1:49 pm

Hi

I have included a link where you can download the log that contains information about uploading a file over 5Gb with the default rclone configuration we have in our Desktop Blomp Go software. Below is my configuration that is also the same as the latest Blomp Go software.

Code: Select all

[remote]
type = swift
user = [email address]
key = ***********
auth = https://authenticate.ain.net
tenant = storage
auth_version = 2
https://www.bigg.ly/downloads/ODhmNzZjN ... BhaS5uZXQ=

Ps: at the bottom of the log contains a failure to upload a file larger than 60 Gb. It is due to I don't have enough space to hold the cache in my C drive.

Please let me know if you have any other questions.

Thank you

BlompTechie

User avatar
BlompTeam
Site Admin
Posts: 359
Joined: Wed Nov 07, 2018 8:14 am

Re: File size limit

Post by BlompTeam » Mon Jun 14, 2021 3:40 pm

BlompTechie wrote:
Mon Jun 14, 2021 1:49 pm
Hi

I have included a link where you can download the log that contains information about uploading a file over 5Gb with the default rclone configuration we have in our Desktop Blomp Go software. Below is my configuration that is also the same as the latest Blomp Go software.

Code: Select all

[remote]
type = swift
user = [email address]
key = ***********
auth = https://authenticate.blomp.com
tenant = storage	
auth_version = 2
https://www.bigg.ly/downloads/ODhmNzZjN ... BhaS5uZXQ=

Ps: at the bottom of the log contains a failure to upload a file larger than 60 Gb. It is due to I don't have enough space to hold the cache in my C drive.

Please let me know if you have any other questions.

Thank you

BlompTechie
BlompTeam

---
Blomp
Blomp Music Video

lloydcia1
Posts: 14
Joined: Mon Jun 07, 2021 2:40 pm

Re: File size limit

Post by lloydcia1 » Wed Jun 16, 2021 5:14 am

Thanks again.

I am getting acceptable transfer speeds now and the files appear to be uploading. However, I am seeing only the 5GB chunks of the file. Am I incorrect in assuming that these chunks should be reconstructed into a whole file and the chunks deleted, rather than remaining 20 separate chunks?

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Wed Jun 16, 2021 12:04 pm

BlompTeam wrote:
Mon Jun 14, 2021 3:40 pm
BlompTechie wrote:
Mon Jun 14, 2021 1:49 pm
Hi

I have included a link where you can download the log that contains information about uploading a file over 5Gb with the default rclone configuration we have in our Desktop Blomp Go software. Below is my configuration that is also the same as the latest Blomp Go software.

Code: Select all

[remote]
type = swift
user = [email address]
key = ***********
auth = https://authenticate.blomp.com
tenant = storage	
auth_version = 2
https://www.bigg.ly/downloads/ODhmNzZjN ... BhaS5uZXQ=

Ps: at the bottom of the log contains a failure to upload a file larger than 60 Gb. It is due to I don't have enough space to hold the cache in my C drive.

Please let me know if you have any other questions.

Thank you

BlompTechie
Ok, it clearly is not working for linux. Like I stated before, initial rclone config, I copied it from blomp datafolder which is equivalent to your rclone.conf.

Also, I did debug back then the app where I could see mount options, according to your log, this is what your app runs:

Code: Select all

/usr/bin/rclone -vvv --config /home/someuser/.config/rclone/rclone.conf \
mount blomp-someuser-remote:someuser@somemail.com /home/someuser/mymountdir \
--volname Blomp --network-mode \
--vfs-cache-mode full --vfs-cache-max-age 1m --dir-cache-time 1m --fast-list --no-checksum --no-modtime --no-seek --transfers 128 --log-file=blomp.log
So, according to that combination, I clearly get on both, windows and linux either, I get the error:

Code: Select all

Failed to copy: Operation forbidden
here a longer log:

Code: Select all

2021/06/16 17:33:04 DEBUG : test/a.iso(0xc00069c340): RWFileHandle.Release
2021/06/16 17:33:04 DEBUG : test/a.iso(0xc00069c340): close: 
2021/06/16 17:33:04 DEBUG : /: Lookup: name="test"
2021/06/16 17:33:04 DEBUG : : Re-reading directory (1m1.992752324s old)
2021/06/16 17:33:04 DEBUG : test/a.iso: vfs cache: setting modification time to 2021-06-16 17:33:04.805281135 +0200 CEST m=+63.272928419
2021/06/16 17:33:04 INFO  : test/a.iso: vfs cache: queuing for upload in 5s
2021/06/16 17:33:04 DEBUG : test/a.iso(0xc00069c340): >close: err=<nil>
2021/06/16 17:33:04 DEBUG : &{test/a.iso (rw)}: >Release: err=<nil>
2021/06/16 17:33:04 DEBUG : /: >Lookup: node=test/, err=<nil>
2021/06/16 17:33:04 DEBUG : test/: Attr: 
2021/06/16 17:33:04 DEBUG : test/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/06/16 17:33:04 DEBUG : test/a.iso: Attr: 
2021/06/16 17:33:04 DEBUG : test/a.iso: >Attr: a=valid=1s ino=0 size=6265860096 mode=-rw-rw-r--, err=<nil>
2021/06/16 17:33:09 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:33:10 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:33:10 ERROR : test/a.iso: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:33:20 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:33:20 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:33:20 ERROR : test/a.iso: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:33:40 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:33:40 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:33:40 ERROR : test/a.iso: vfs cache: failed to upload try #3, will retry in 40s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:34:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:34:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:34:20 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:34:21 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:34:21 ERROR : test/a.iso: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:35:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:35:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:35:41 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:35:41 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:35:41 ERROR : test/a.iso: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:36:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:36:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:37:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:37:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:38:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:38:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:38:21 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:38:21 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:38:21 ERROR : test/a.iso: vfs cache: failed to upload try #6, will retry in 5m0s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:39:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:39:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:40:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:40:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:41:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:41:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:42:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:42:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:43:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:43:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:43:21 DEBUG : test/a.iso: vfs cache: starting upload
2021/06/16 17:43:22 ERROR : test/a.iso: Failed to copy: Operation forbidden
2021/06/16 17:43:22 ERROR : test/a.iso: vfs cache: failed to upload try #7, will retry in 5m0s: vfs cache: failed to transfer file from cache to remote: Operation forbidden
2021/06/16 17:44:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:44:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:45:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:45:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:46:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:46:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
2021/06/16 17:47:02 DEBUG : vfs cache RemoveNotInUse (maxAge=60000000000, emptyOnly=false): item test/a.iso not removed, freed 0 bytes
2021/06/16 17:47:02 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 5.836G (was 5.836G)
By that, I claim that those settings are not working. According to your settings it works. I also tried ai.net and blomp.com auth urls, no difference. Blomp app uses ai.net, where on dashboard it says blomp.com, why is blomp using ai.net and not blomp and will ai.net one be outdated?

By that, my conclusion is that you have to fix either my user or your backend for the type of user (whichever I am).

So, to go step ahead, I can resolve "Failed to copy: Operation forbidden" by adding --swift-chunk-size 1P (or add to remote config chunk_size = 1P).

With that workaround, I do not get anymore operation forbidden, however, I technically disable swift-chunker by that and no files will be chunked which works until max allowed size for swift is reached, which is actually 5G, but on Blomp 5G is too big and upload will fail, few bytes less than 5G already works.

With that, I clearly can state that chunker is not working for me on your side.

Workaround to upload also bigger files than 5G is to use separate chunker overlay like I pointed few times.

Here is small overview:

- with your config and mount line: no files over 5G can be copied (Error: Failed to copy: Operation forbidden)
- with disabled chunker (my workaround): no files over 5G can be copied (this time error is different: Failed to copy: Too Large Object)
- with disabled chunker and additional chunker overlay for 5G (<5G): no restrictions, all files can be uploaded and only files over 5G are chunked.

As you see, I am still forced to use it. Here are some drawbacks for using additional chunker overlay:
- On Blomp dashboard it does not show as single file, one has to mount chunker overlay to see one file.
- Chunker overlay files can not be linked (you can not create share link of the file for chunker/crypt overlay

I believed that using renaming makes it slow for overlay, however, I inspected browser upload which indeed also chunks the file and then it takes some amount of time before next segment is uploaded, this time is same, in my case it takes longer as it is 5G chunk where on browser it chunkes to 1G and therefore moves quicker.

As there are other users who reported the same, I guess we all who report have no proper permissions, normally this leads to think that it has to do with local configs, but it does not, all permissions on my side seem to be proper (unless you post correct values stating that my are wrong). Adding --uid 1000 --gid 1000 as well as umask does not help for my user or my demoaccounts, they all suffer on the same error and none of them is capable uploading files over 5G.

I also could see that you used ai.net email addrress, maybe that is where it differs, maybe your account has some more/proper permission.

Please recheck my user and my permissions on your backend

EDIT: additioinal note which I forgot, maybe I am wrong, but as far as I believe, --fast-list has no effect when mounting with rclone.
Last edited by demoaccount on Wed Jun 16, 2021 12:49 pm, edited 2 times in total.

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Wed Jun 16, 2021 12:41 pm

lloydcia1 wrote:
Wed Jun 16, 2021 5:14 am
Thanks again.

I am getting acceptable transfer speeds now and the files appear to be uploading. However, I am seeing only the 5GB chunks of the file. Am I incorrect in assuming that these chunks should be reconstructed into a whole file and the chunks deleted, rather than remaining 20 separate chunks?
You need to mount your chunker overlay, then they all show as one file.

I can confirm that all combinations including crypt work pretty well. I am using blomp mainly as archive, but tested it with kodi and some pvr devices where watching movies, pictures etc.. works perfectly.

If you copy a folder with thousands of small files, consider using high values for transfers and checkers, I used with blomp --transfers 1024 and --checkers 1024, which speeds up the upload time drastically, on other side, if you copy large files, better stick to smaller values, blompSupport uses in above config 128, I use 32 per default, if I need to copy huge folder with many files, I normally do it over terminal with rclone, but you can mount it several times as much as you need.

lloydcia1
Posts: 14
Joined: Mon Jun 07, 2021 2:40 pm

Re: File size limit

Post by lloydcia1 » Wed Jun 16, 2021 2:13 pm

If you're referring to blomp-chunker:, it is mounted.

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Wed Jun 16, 2021 2:46 pm

lloydcia1 wrote:
Wed Jun 16, 2021 2:13 pm
If you're referring to blomp-chunker:, it is mounted.
yes, I do. If it is mounted, you should see it as one single file.

demoaccount
Posts: 42
Joined: Sun May 23, 2021 3:41 pm

Re: File size limit

Post by demoaccount » Wed Jun 16, 2021 3:53 pm

Just as info:
- Blomp Max file size (swift): 5242879K
- Swift default max chunk size: 5G which is 5242880K

@BlompSupport
using chunker without rename is auite faster than uploading over dashboard, and all hashes are the same which is very reliable.

What for is 1K used and why 5242880K is not working, do you know?

Post Reply