Go

关于golang 的readall清空reader

2018-04-23  本文已影响0人  svenke

问题

   body, err := ioutil.ReadAll(c.Request().Body) 
   代码执行完成后,c.Request().Body 内容变成了空,这是为什么?

代码分析

step1 调用内部函数,每次至少读512字节(怎么定的呢?)

可以注意到底层调用了buffer的ReadFrom函数

// readAll reads from r until an error or EOF and returns the data it read
// from the internal buffer allocated with a specified capacity.
func readAll(r io.Reader, capacity int64) (b []byte, err error) {
    buf := bytes.NewBuffer(make([]byte, 0, capacity))
    // If the buffer overflows, we will get bytes.ErrTooLarge.
    // Return that as an error. Any other panic remains.
    defer func() {
        e := recover()
        if e == nil {
            return
        }
        if panicErr, ok := e.(error); ok && panicErr == bytes.ErrTooLarge {
            err = panicErr
        } else {
            panic(e)
        }
    }()
    _, err = buf.ReadFrom(r)
    return buf.Bytes(), err
}

// ReadAll reads from r until an error or EOF and returns the data it read.
// A successful call returns err == nil, not err == EOF. Because ReadAll is
// defined to read from src until EOF, it does not treat an EOF from Read
// as an error to be reported.
func ReadAll(r io.Reader) ([]byte, error) {
    return readAll(r, bytes.MinRead)
}

step2 buffer readFrom实现分析

Buffer定义,buf:存放的数据,buf中off到len(buf)之间的数据为有效数据,其余数据为0,读的时候从buf[off]开始,写是在buf[len(buf)]。

// A Buffer is a variable-sized buffer of bytes with Read and Write methods.
// The zero value for Buffer is an empty buffer ready to use.
type Buffer struct {
    buf       []byte   // contents are the bytes buf[off : len(buf)]
    off       int      // read at &buf[off], write at &buf[len(buf)]
    bootstrap [64]byte // memory to hold first slice; helps small buffers avoid allocation.
    lastRead  readOp   // last read operation, so that Unread* can work correctly.
}
// ReadFrom reads data from r until EOF and appends it to the buffer, growing
// the buffer as needed. The return value n is the number of bytes read. Any
// error except io.EOF encountered during the read is also returned. If the
// buffer becomes too large, ReadFrom will panic with ErrTooLarge.
func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error) {
    b.lastRead = opInvalid
    // If buffer is empty, reset to recover space.
    if b.off >= len(b.buf) {
        b.Truncate(0)
    }
    for {
        //buff 末尾使用的空间不足时,扩容
        if free := cap(b.buf) - len(b.buf); free < MinRead {
            // not enough space at end,buffer正常情况下是直接尾部写入数据,当发现尾部空间不足时,就需要调整buffer存储形式,将数据区buf[off]到buf[len(buf)]对齐到头部,攒出尾部空间来继续存储。

            newBuf := b.buf 
            if b.off+free < MinRead {  //发现可用的空间不够用时,就需要扩容,扩容办法是原buf cap的两倍+最小读取数据大小
                // not enough space using beginning of buffer;
                // double buffer capacity
                newBuf = makeSlice(2*cap(b.buf) + MinRead)
            }
            copy(newBuf, b.buf[b.off:])//将buf数据区copy到临时buf
            b.buf = newBuf[:len(b.buf)-b.off]//再从临时buf把数据copy回去,完成对齐头部
            b.off = 0
        }
        m, e := r.Read(b.buf[len(b.buf):cap(b.buf)])//传入buf 的长度与cap之间的空间给reader
        b.buf = b.buf[0 : len(b.buf)+m]//buf取到真实读取的字节位置
        n += int64(m)//读取的总字节数增加
        if e == io.EOF {//读到eof标记时停止
            break
        }
        if e != nil {
            return n, e
        }
    }
    return n, nil // err is EOF, so return nil explicitly
}

step3 io.Reader 的Read实现;以c.Request().Body为例

其中body是这么定义的,是封装了Reader和Closer的interface

Body io.ReadCloser

那么body的reader到底是什么呢?body是在Request结构体中定义的,内容是请求的body体,因此连接建立起来后,body会被赋值,沿着这个思路,找到准备body的reader代码的地方,详细查找路基如下:
【http启动服务】func (srv *Server) Serve(l net.Listener) =>【建立连接】go c.serve(ctx)=>【读取request】c.readRequest(ctx)=>【读取request详细信息】readRequest(b *bufio.Reader, deleteHostHeader bool)=>【解析http协议,读取信息】readTransfer(msg interface{}, r *bufio.Reader)

// Prepare body reader. ContentLength < 0 means chunked encoding
    // or close connection when finished, since multipart is not supported yet
    switch {
    case chunked(t.TransferEncoding):
        if noResponseBodyExpected(t.RequestMethod) {
            t.Body = NoBody
        } else {
            t.Body = &body{src: internal.NewChunkedReader(r), hdr: msg, r: r, closing: t.Close}
        }
    case realLength == 0:
        t.Body = NoBody
    case realLength > 0:
        t.Body = &body{src: io.LimitReader(r, realLength), closing: t.Close}
    default:
        // realLength < 0, i.e. "Content-Length" not mentioned in header
        if t.Close {
            // Close semantics (i.e. HTTP/1.0)
            t.Body = &body{src: r, closing: t.Close}
        } else {
            // Persistent connection (i.e. HTTP/1.1)
            t.Body = NoBody
        }
    }

至此,我们发现body是一个bufio.Reader,其定义如下

type Reader struct {
    buf          []byte //缓冲区
    rd           io.Reader // reader provided by the client
    r, w         int       // buf read and write positions
    err          error
    lastByte     int
    lastRuneSize int
}

在server.go中,我们注意到如下初始化,也就是说body是一个带缓冲区的io reader,并且connReader作为数据源的reader

c.r = &connReader{conn: c}
c.bufr = newBufioReader(c.r)

关键!从缓冲区读取数据区,并将buf的读off加n,也就是将buf中已读的数据清空

// Read reads data into p.
// It returns the number of bytes read into p.
// The bytes are taken from at most one Read on the underlying Reader,
// hence n may be less than len(p).
// At EOF, the count will be zero and err will be io.EOF.
func (b *Reader) Read(p []byte) (n int, err error) {
    n = len(p)
    if n == 0 {
        return 0, b.readErr()
    }
    if b.r == b.w {
        if b.err != nil {
            return 0, b.readErr()
        }
        if len(p) >= len(b.buf) {
            // Large read, empty buffer.
            // Read directly into p to avoid copy.
            n, b.err = b.rd.Read(p)
            if n < 0 {
                panic(errNegativeRead)
            }
            if n > 0 {
                b.lastByte = int(p[n-1])
                b.lastRuneSize = -1
            }
            return n, b.readErr()
        }
        // One read.
        // Do not use b.fill, which will loop.
        b.r = 0
        b.w = 0
        n, b.err = b.rd.Read(b.buf)////buf为空时,直接从conn的reader中读取,不涉及r、w的变化,buf还是空
        if n < 0 {
            panic(errNegativeRead)
        }
        if n == 0 {
            return 0, b.readErr()
        }
        b.w += n
    }

    // copy as much as we can
    n = copy(p, b.buf[b.r:b.w])//从缓冲区读取数据区
    b.r += n//将buf的读off加n,也就是将buf中已读的数据清空
    b.lastByte = int(b.buf[b.r-1])
    b.lastRuneSize = -1
    return n, nil
}

结论

ioutil.ReadAll(c.Request().Body) 最终会调用bufio.Reader 的read方法来读取数据,buf被读取后,数据会被清空,所以 ioutil.ReadAll(c.Request().Body) 再次调用时,也读不到数据啦~

上一篇 下一篇

猜你喜欢

热点阅读