1 / 15

Evaluation Methodology

Evaluation Methodology. Date: 2013-09-16. Authors:. Outline. An evaluation methodology document for HEW is required in order to assess gains relative to 11ac. We provide here a description of several topics: Simulation methodology Metrics Traffic models.

enya
Télécharger la présentation

Evaluation Methodology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation Methodology Date:2013-09-16 Authors: Ron Porat, Broadcom

  2. Outline • An evaluation methodology document for HEW is required in order to assess gains relative to 11ac. • We provide here a description of several topics: • Simulation methodology • Metrics • Traffic models Ron Porat, Broadcom

  3. Simulation Methodology - General Concept • We propose to have three tools to assess proposed HEW techniques performance and gain relative to 11ac, each having its own advantages : • PER simulations – typically used for new PHY features for assessing point to point performance • PHY System simulations – provide system-wise (multi-BSS) performance assessment with emphasis on PHY abstraction accuracy and simplified MAC • MAC System simulations - provides system-wise (multi-BSS) performance assessment with emphasis on MAC accuracy and simplified PHY • While PER simulations have been used extensively in the past, system simulations have not been used nearly as much. However, based on the discussions in HEW so far, the main effort revolves around techniques for improving multi-BSS gain and not higher point to point throughput. Hence the need for system simulations. • PHY/MAC system simulations may be used separately in order to: • Simplify (using abstraction) some of the MAC/PHY details respectively • Speed up development (by reducing dependency of PHY on MAC and vice versa) • Improve insight as to the specific reason for performance gains/losses (a more difficult thing to assess in multi-STA multi-BSS environments that include all possible PHY and MAC details). • Some proposed techniques may not require all PHY/MAC details in order to evaluate gains but are nevertheless used over the same simulation scenarios as defined in [10][11] Ron Porat, Broadcom

  4. System Simulation - General Description • A system simulation is comprised of multiple drops and multiple TXOP. • A drop is defined by a known AP and STA locations. Simulating multiple drops is important in order to randomize STA location to better assess issues such as OBSS interference and intra-BSS throughput (as these issues are highly dependent on the relative path loss between AP and STA). • In a TXOP a set of transmissions occur. Multiple TXOP (with typical aggregate duration 1-10[sec] beyond a warm-up time) are required to assess performance of a given set of APs and STAs configuration. Each BSS may have different start time, duration and end time for its TXOP but time alignment (start, duration, end) of the TXOP in the system may be an outcome of a proposed MAC protocol. • A’warm-up’ period may be used for some parameters to converge: • MCS selection based on the past • Offered load - if all flows start at T0, then the offered load goes from 0 to X instantaneously, and over some simulation time that follows, the backoff mechanism will effectively reduce the total offered load by increasing CW at nodes, until the offered load is at Y < X Ron Porat, Broadcom

  5. Cont. • General simulation structure: • For drop=1:N { • APs and STA are dropped (randomly or in fixed locations) in a given area following certain placement rules (specifics depend on the scenario). • If simulation time is an issue, multiple drops’ locations can be agreed beforehand in order to minimize the number of drops and enable multiple companies’ results comparison. • STAs are associated with APs (typically based on path loss but could use other rules if some AP are not be available (e.g. belonging to a different apartment or operator) in some scenarios. • For TXOP=1:M { • Note - For MAC simulations, one can count time, ensuring that enough time has passed to see M TXOP • Perform transmissions consistent with all the parameters defined and collect metrics. Parameters to be defined: • For DL and UL – channel access rules: EDCA, centralized control, restricted access windows etc.. • For DL (and UL if centralized control is proposed): how the AP allocates air time to users -e.g. proportional fair, round robin. • CCA rules • Various other EDCA parameters as specified in the MAC simulations section • } • } Ron Porat, Broadcom

  6. Simulation Methodology I - PER Simulations • PHY PER simulations are used to verify point to point performance or aspects that need this type of simulation, especially new PHY features or preamble performance. • PHY impairments such as PA non-linearity, phase noise, channel estimation error, non-linear receivers are more readily incorporated into PER simulations and may be needed if some techniques are adversely affected by those impairments [6][9]. • Other impairments such as the impact of OBSS interference or inter-symbol interference should also be tested by PER simulations before being abstracted in a system simulation. • PER simulations should also be used to investigate potential error floors that are not well captured by system simulations due to PHY abstraction inaccuracies. Ron Porat, Broadcom

  7. Simulation Methodology II - PHY System Sims • As discussed the emphasis here is on more accurate representation of the PHY using PHY abstraction • For simplicity one might consider an assumption of time aligned TXOPs. • PHY parameters that need to be defined per AP and per STA: • Transmission BW (could be different in different BSS) • Number of antennas (could be different for different STA) • Transmission scheme- SU OL, SU BF, MU • Channel model (multipath fading, Doppler, path loss, shadowing etc..) • Link adaptation needs to be described as well – • MCS choice (genie could be used to assess upper performance bound but not realistic performance). • Transmission mode selection SU OL vs. BF vs. MU should be described (transmission mode can be fixed or dynamic). • A subset of the full set of MAC features can be assumed: • E.g. CCA levels and RTS-CTS can be specified for the purpose of determining which transmission can simultaneously occur in multiple BSS • No need to simulate management frames but rather a fixed amount of overhead can be assumed Ron Porat, Broadcom

  8. PHY Abstraction • The objective of PHY abstraction is to accurately predict PER simulation results in a computationally simple way to enable running system simulations in a timely manner. • The underlying principle is to calculate an effective average SINR (SINReff ) in a given OFDM symbol. This quantity then acts as a link between AWGN PER and multipath channel PER for a given coding type, block size and MCS level. • Effective SINR (SINReff) is typically calculated as follows where SINRn is the post processing SINR at the n-th subcarrier, N is the number of subcarriers in a coded block and Φ is a mapping function. • Several mapping functions can be used such as Constrained Capacity, EESM, MMIB, etc. [12] Ron Porat, Broadcom

  9. Cont. • A general description is as follows: • Start from an agreed upon per-MCS required SNR in AWGN assuming e.g. 1000bytes SISO 10% PER point. • With one receive antenna: • Compute SINR per tone – the ‘S’ term is a function of the Tx power and channel. The ‘I’ term is due to OBSS, intra-BSS interference or MU-MIMO related interference. Note that ‘I’ could vary during a packet due to shorter interfering packet than the desired packet or start of new interfering packet midway through the desired packet. • If constraint capacity is used, calculate the per-tone capacity log2(1+SINR), this could be constrained to 256QAM capacity, and average across all data tones used for transmission. If MMIB is used, calculate the average per-bit capacity as described in [13] If RBIR is used, calculate the symbol information as described in [14] • From 2, derive the averaged SINR per tone. • Transform back to MCS using the AWGN MCS table • With multiple receive antennas: • SINR should reflect the receive combining output from all antennasand the combining method should be indicated • For MIMO simulation, linear per-stream SINR can be computed. Ron Porat, Broadcom

  10. Simulation Methodology III - MAC System Sims • As discussed the emphasis here is on more accurate representation of the MAC • MAC parameters that need to be described: • Beacon periodicity • Aggregation policy • Usage of RTS-CTS or CTS2SELF • EDCA parameters (AC, CWmin, CWmax, AIFSN) • CSMA backoff procedures, energy and preamble detection • Basic rate set • If other MAC schemes are proposed they need to be described (e.g. RAW as defined in 11ah or HCCA) • Link adaptation needs to be described – • MCS choice (genie could be used to assess upper performance bound but not realistic performance). • Frequency re-use needs to be described • PHY parameters may be simplified to a SISO configuration running over AWGN. • Accurate path loss and penetration should be modeled • In the event that some techniques require both accurate MAC and PHY details to be simulated, this simulation can be augmented by adding more detailed PHY abstraction used for simulation methodology II to provide a complete system simulation tool Ron Porat, Broadcom

  11. Simulation Methodology Choice • Proponents of different techniques should also provide justification for their proposed simulation tool used to justify the technique’s gains. • Examples: • PHY PER simulation: • New PHY FFT size – as was done in 11ac (256,512) and 11ah (32) a PER simulation is typically sufficient in order to decide the number of pilots, interleaver parameters etc. • Preamble performance • Implementation losses of current and new PHY modes. • PHY System simulation: • Impact of number of antennas on multi-BSS performance • Impact of PHY techniques in the context of multi-BSS • Impact of frequency re-use in multi-BSS • Impact of CCA levels on PHY performance • MAC System simulation: • Impact of MAC scheduler – EDCA vs. RAW (as proposed in 11ah) vs. HCCA vs. other techniques • Impact of frequency re-use in multi-BSS • Impact of CCA levels Ron Porat, Broadcom

  12. Metrics • For PER simulations the typical metric is dB gain/loss in waterfall curves. • For system simulations we propose to look at several metrics [2]-[9]: • Aggregate area throughput [bps/m2] - this metric directly relates to the average throughput per BSS and can more accurately compare different deployment densities and heterogeneous deployments. For example: • How is area throughput impacted as a function of inter AP distance and number of STA? • How is area throughput impacted with flat deployments vs. heterogeneous? • Average per-STA throughput in all participating BSS • 5% point in the throughput CDF curve – this metric measures cell edge performance • More information may be provided such as the entire CDF curve and MCS histogram [3] Ron Porat, Broadcom

  13. Traffic Models • Two main traffic models may be simulated: • Full buffer – users always have DATA to send • Poisson arrival – more realistic traffic model emulating more realistic interference scenario • Packet size – mix of small and large packets should be evaluated in order to test realistic assumptions on system performance • Higher layer QoS metrics may be developed [4] • Video transmission specific metrics may be developed [5] • Other specific types of traffic models may be used [6][7] Ron Porat, Broadcom

  14. Straw Poll • Do you agree to use contribution 13/1051 as a baseline evaluation methodology? • Yes • No • Abs Ron Porat, Broadcom

  15. [1] 11-13-0657-02-0hew-hew-sg-usage-models-and-requirements-liaison-with-wfa [2] 11-13-0486-01-0hew-metrics-targets [3] 11-13-0847-01-0hew-evaluation-criteria-and-simulation-scenarios [4] 11-13-0869-00-0hew-simulation-scenarios-and-metrics-for-hew [5] 11-13-0850-00-0hew-quantitative-qoe-requirements-for-hew [6] 11-13-0722-01-0hew-hew-evaluation-methodology [7] 11-13-0723-00-0hew-hew-sg-evaluation-methodology-overview [8] 11-13-0786-00-0hew-hew-sls-methodology [9] 11-13-0837-00-0hew-considerations-on-hew-evaluation-methodology [10] 11-13-1000-00-0hew-simulation-scenarios [11] 11-13-1001-00-0hew-simulation-scenarios-document-template [12] IEEE 802.16m-08/004r5, IEEE 802.16m Evaluation Methodology Document (EMD) Section 4 http://ieee802.org/16/tgm/core.html#08_004 [13] 11-13-1059-00-0hew-PHY abstraction for HEW evaluation methodology [14] 11-13-1131-00-0hew-PHY abstraction for HEW system level simulation References Ron Porat, Broadcom

More Related