OutOfMemoryException podczas wysyłania dużego pliku 500MB przy użyciu FILESTREAM ASPNET

Używam Filestream do odczytu dużego pliku (>500 MB) i otrzymuję OutOfMemoryException.

Używam Asp.net ,. NET 3.5, win2003, iis 6.0

Chcę to w mojej aplikacji:

Odczyt danych z Oracle

Rozpakuj plik za pomocą FileStream i BZip2

Odczytaj plik nieskompresowany i wyślij go do asp.net strona do pobrania.

Kiedy czytam Plik z dysku, nie działa !!! i wynoś się z pamięci...

. Mój kod to:

using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read)) 
        { 
          byte[] b2 = ReadFully(fs3, 1024); 
        } 

 // http://www.yoda.arachsys.com/csharp/readbinary.html
 public static byte[] ReadFully(Stream stream, int initialLength) 
  { 
    // If we've been passed an unhelpful initial length, just 
    // use 32K. 
    if (initialLength < 1) 
    { 
      initialLength = 32768; 
    } 

    byte[] buffer = new byte[initialLength]; 
    int read = 0; 

    int chunk; 
    while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0) 
    { 
      read += chunk; 

      // If we've reached the end of our buffer, check to see if there's 
      // any more information 
      if (read == buffer.Length) 
      { 
        int nextByte = stream.ReadByte(); 

        // End of stream? If so, we're done 
        if (nextByte == -1) 
        { 
          return buffer; 
        } 

        // Nope. Resize the buffer, put in the byte we've just 
        // read, and continue 
        byte[] newBuffer = new byte[buffer.Length * 2]; 
        Array.Copy(buffer, newBuffer, buffer.Length); 
        newBuffer[read] = (byte)nextByte; 
        buffer = newBuffer; 
        read++; 
      } 
    } 
    // Buffer is now too big. Shrink it. 
    byte[] ret = new byte[read]; 
    Array.Copy(buffer, ret, read); 
    return ret; 
  } 

Teraz określam mój problem lepiej.

Rozpakuj plik za pomocą FileStream i BZip2 jest OK, wszystko jest w porządku.

Problem jest następujący:

Odczytaj plik fat big na dysku (>500 MB) w bajcie [] i wyślij bajty do odpowiedzi (asp.net) do pobrania.

Kiedy używać

Http://www.yoda.arachsys.com/csharp/readbinary.html

public static byte[] ReadFully

Dostaję błąd: OutOfMemoryException...

If better Buforedstream than Stream (FileStream, MemoryStream, ...) ??

Za pomocą BufferedStream, Czy mogę odczytać duży plik 700 MB ?? (dowolny przykładowy kod źródłowy wykorzystujący BufferedStream do pobrania dużego pliku)

Myślę, że to jest pytanie: nie " jak odczytać plik 500mb do pamięci?", Ale " jak wysłać duży plik do strumienia odpowiedzi ASPNET?"

Znalazłem ten kod przez Cheese:

using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))  
{  
   Response.BufferOutput= false;   // to prevent buffering 
   byte[] buffer = new byte[1024]; 
   int bytesRead = 0; 
   while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)  
   { 
       Response.OutputStream.Write(buffer, 0, bytesRead); 
   } 
}
Czy to dobry kod ?? jakieś ulepszenia dla wysokiej wydajności ??

A collegue say me, use

Response.TransmitFile(filePath);

Teraz kolejne pytanie, lepiej transmitować plik lub kod przez Cheese ??

Wiele lat temu, w msdn magazine pojawia się świetny artykuł o tym, ale nie mogę uzyskać dostępu http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/,

Update : możesz uzyskać dostęp za pomocą webarchive w linku: https://web.archive.org/web/20070627063111/http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/

Jakieś sugestie, uwagi, przykładowe źródło kodu??

Author: Kiquenet, 2010-07-29

3 answers

Stworzyłem stronę pobierania, która pozwala użytkownikowi pobierać do 4gb (może być więcej) kilka miesięcy temu. Oto mój roboczy fragment:

  private void TransmitFile(string fullPath, string outFileName)
    {
        System.IO.Stream iStream = null;

        // Buffer to read 10K bytes in chunk:
        byte[] buffer = new Byte[10000];

        // Length of the file:
        int length;

        // Total bytes to read:
        long dataToRead;

        // Identify the file to download including its path.
        string filepath = fullPath;

        // Identify the file name.
        string filename = System.IO.Path.GetFileName(filepath);

        try
        {
            // Open the file.
            iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
                        System.IO.FileAccess.Read, System.IO.FileShare.Read);


            // Total bytes to read:
            dataToRead = iStream.Length;

            Response.Clear();
            Response.ContentType = "application/octet-stream";
            Response.AddHeader("Content-Disposition", "attachment; filename=" + outFileName);
            Response.AddHeader("Content-Length", iStream.Length.ToString());

            // Read the bytes.
            while (dataToRead > 0)
            {
                // Verify that the client is connected.
                if (Response.IsClientConnected)
                {
                    // Read the data in buffer.
                    length = iStream.Read(buffer, 0, 10000);

                    // Write the data to the current output stream.
                    Response.OutputStream.Write(buffer, 0, length);

                    // Flush the data to the output.
                    Response.Flush();

                    buffer = new Byte[10000];
                    dataToRead = dataToRead - length;
                }
                else
                {
                    //prevent infinite loop if user disconnects
                    dataToRead = -1;
                }
            }
        }
        catch (Exception ex)
        {
            throw new ApplicationException(ex.Message);
        }
        finally
        {
            if (iStream != null)
            {
                //Close the file.
                iStream.Close();
            }
            Response.Close();
        }
    }
 16
Author: Pavel Morshenyuk,
Warning: date(): Invalid date.timezone value 'Europe/Kyiv', we selected the timezone 'UTC' for now. in /var/www/agent_stack/data/www/doraprojects.net/template/agent.layouts/content.php on line 54
2010-07-29 13:35:45

Nie trzeba trzymać całego pliku w pamięci, wystarczy go odczytać i zapisać do strumienia odpowiedzi w pętli.

 2
Author: Giorgi,
Warning: date(): Invalid date.timezone value 'Europe/Kyiv', we selected the timezone 'UTC' for now. in /var/www/agent_stack/data/www/doraprojects.net/template/agent.layouts/content.php on line 54
2010-07-29 13:22:45

Istnieje więcej niż jedno rozwiązanie

1-Używanie RecyclableMemoryStream zamiast MemoryStream solution

Więcej o RecyclableMemoryStream przeczytasz tutaj : http://www.philosophicalgeek.com/2015/02/06/announcing-microsoft-io-recycablememorystream/

Https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

2-Używanie MemoryTributary zamiast MemoryStream

Możesz przeczytać więcej o MemoryTributary tutaj :

Https://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream?msg=5257615#xx5257615xx

    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Runtime.InteropServices;

   namespace LiquidEngine.Tools
       {
/// <summary>
/// MemoryTributary is a re-implementation of MemoryStream that uses a dynamic list of byte arrays as a backing store, instead of a single byte array, the allocation
/// of which will fail for relatively small streams as it requires contiguous memory.
/// </summary>
public class MemoryTributary : Stream       /* http://msdn.microsoft.com/en-us/library/system.io.stream.aspx */
{
    #region Constructors

    public MemoryTributary()
    {
        Position = 0;
    }

    public MemoryTributary(byte[] source)
    {
        this.Write(source, 0, source.Length);
        Position = 0;
    }

    /* length is ignored because capacity has no meaning unless we implement an artifical limit */
    public MemoryTributary(int length)
    {
        SetLength(length);
        Position = length;
        byte[] d = block;   //access block to prompt the allocation of memory
        Position = 0;
    }

    #endregion

    #region Status Properties

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    #endregion

    #region Public Properties

    public override long Length
    {
        get { return length; }
    }

    public override long Position { get; set; }

    #endregion

    #region Members

    protected long length = 0;

    protected long blockSize = 65536;

    protected List<byte[]> blocks = new List<byte[]>();

    #endregion

    #region Internal Properties

    /* Use these properties to gain access to the appropriate block of memory for the current Position */

    /// <summary>
    /// The block of memory currently addressed by Position
    /// </summary>
    protected byte[] block
    {
        get
        {
            while (blocks.Count <= blockId)
                blocks.Add(new byte[blockSize]);
            return blocks[(int)blockId];
        }
    }
    /// <summary>
    /// The id of the block currently addressed by Position
    /// </summary>
    protected long blockId
    {
        get { return Position / blockSize; }
    }
    /// <summary>
    /// The offset of the byte currently addressed by Position, into the block that contains it
    /// </summary>
    protected long blockOffset
    {
        get { return Position % blockSize; }
    }

    #endregion

    #region Public Stream Methods

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        long lcount = (long)count;

        if (lcount < 0)
        {
            throw new ArgumentOutOfRangeException("count", lcount, "Number of bytes to copy cannot be negative.");
        }

        long remaining = (length - Position);
        if (lcount > remaining)
            lcount = remaining;

        if (buffer == null)
        {
            throw new ArgumentNullException("buffer", "Buffer cannot be null.");
        }
        if (offset < 0)
        {
            throw new ArgumentOutOfRangeException("offset",offset,"Destination offset cannot be negative.");
        }

        int read = 0;
        long copysize = 0;
        do
        {
            copysize = Math.Min(lcount, (blockSize - blockOffset));
            Buffer.BlockCopy(block, (int)blockOffset, buffer, offset, (int)copysize);
            lcount -= copysize;
            offset += (int)copysize;

            read += (int)copysize;
            Position += copysize;

        } while (lcount > 0);

        return read;

    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        switch (origin)
        {
            case SeekOrigin.Begin:
                Position = offset;
                break;
            case SeekOrigin.Current:
                Position += offset;
                break;
            case SeekOrigin.End:
                Position = Length - offset;
                break;
        }
        return Position;
    }

    public override void SetLength(long value)
    {
        length = value;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        long initialPosition = Position;
        int copysize;
        try
        {
            do
            {
                copysize = Math.Min(count, (int)(blockSize - blockOffset));

                EnsureCapacity(Position + copysize);

                Buffer.BlockCopy(buffer, (int)offset, block, (int)blockOffset, copysize);
                count -= copysize;
                offset += copysize;

                Position += copysize;

            } while (count > 0);
        }
        catch (Exception e)
        {
            Position = initialPosition;
            throw e;
        }
    }

    public override int ReadByte()
    {
        if (Position >= length)
            return -1;

        byte b = block[blockOffset];
        Position++;

        return b;
    }

    public override void WriteByte(byte value)
    {
        EnsureCapacity(Position + 1);
        block[blockOffset] = value;
        Position++;
    }

    protected void EnsureCapacity(long intended_length)
    {
        if (intended_length > length)
            length = (intended_length);
    }

    #endregion

    #region IDispose

    /* http://msdn.microsoft.com/en-us/library/fs2xkftw.aspx */
    protected override void Dispose(bool disposing)
    {
        /* We do not currently use unmanaged resources */
        base.Dispose(disposing);
    }

    #endregion

    #region Public Additional Helper Methods

    /// <summary>
    /// Returns the entire content of the stream as a byte array. This is not safe because the call to new byte[] may 
    /// fail if the stream is large enough. Where possible use methods which operate on streams directly instead.
    /// </summary>
    /// <returns>A byte[] containing the current data in the stream</returns>
    public byte[] ToArray()
    {
        long firstposition = Position;
        Position = 0;
        byte[] destination = new byte[Length];
        Read(destination, 0, (int)Length);
        Position = firstposition;
        return destination;
    }

    /// <summary>
    /// Reads length bytes from source into the this instance at the current position.
    /// </summary>
    /// <param name="source">The stream containing the data to copy</param>
    /// <param name="length">The number of bytes to copy</param>
    public void ReadFrom(Stream source, long length)
    {
        byte[] buffer = new byte[4096];
        int read;
        do
        {
            read = source.Read(buffer, 0, (int)Math.Min(4096, length));
            length -= read;
            this.Write(buffer, 0, read);

        } while (length > 0);
    }

    /// <summary>
    /// Writes the entire stream into destination, regardless of Position, which remains unchanged.
    /// </summary>
    /// <param name="destination">The stream to write the content of this stream to</param>
    public void WriteTo(Stream destination)
    {
        long initialpos = Position;
        Position = 0;
        this.CopyTo(destination);
        Position = initialpos;
    }

    #endregion
}

}

 0
Author: Ebram,
Warning: date(): Invalid date.timezone value 'Europe/Kyiv', we selected the timezone 'UTC' for now. in /var/www/agent_stack/data/www/doraprojects.net/template/agent.layouts/content.php on line 54
2018-07-25 19:21:43